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The(;Qgj^jQpment of a Process Evaluation Instrument. 

Linda Proudfit 

• .One area of evaluation which has received much attention is that 
of classroom testing. ' Traditional testing instruments, normally stan- 
dardized or criterion-referenced, have been developed to assess final ' 
outcomes. What these instruments do not evaluate are the processes 
which children use to make decisions or to reach' conclusions. The area 

^f mathematical problem solving is one in which the evaluation of pro- 
cesses is of;particular impo.rtance. ' y 

Little has been done to develop instruments wh4^^valuate-comple/ 
behaviors such as problem-solving jjrocesses. In 1^61j.?the National - 

Council of Teachers of Mathematics noted their concern about thj/^ln- 

ad6quafcy. 

^ The committee vyouVl have liked to include material on the 

appr.aisal of mental processes in the learning of. mathematics. 
As teachers of mathematics, we ar* deeply concerned about ^ 

, .developing skill in productive thinking. Too often, many t)f ■ . • 
. us firtd jirselves knowing little about the relations betWeen -. - 
fw students and the' thought processes / 

that add to those solutions. However, tests for appraising / 
higtier. mental processes such as concept formation, problem / 
solving; and creative thinking in mathematics do not exist. / 
(Johnson, 1961 , pp. '2-3) v . / 

More rebeVitly^the National Advisory Committee on Mathematics Education 
CI 975)^ Itated .that evaluation of e/forts to apply mathematics to4 
variety of problemV i s needed. Kilpatrick (1976) surmiarized t^ie situ- 
ation as follows: - , ' / ' 

A common complaint in the literature on evaluation in4athe- 
matics education has been..that we lack instruments to, measure 
such constructs as problem-solving ability and "creativity in 
mathematics. Certainly if instrumentsVwere available to mea--. ' 
sure these constructs they Could bfe, used by teachers to mea- 
sure ayhild's mathematical' development as well ^s'^-By~evalu- 
atprs of curriculum projects to measure how sucdessful- the- ■ • 
' P>"oject hid beejn in. developing* these qualities/in children. 
(P- 18) ' ■ V / • , . 



The purpose of .this report is to describe the efforts of the <l 
Mathematical Problem Solving Project to develop a paperrand-pencil . 
Instrument which coiild be used to assess processes used by children in. 
attempting to solve nonroutine mathematical problems. The term "process" 
is used here in a ratheV broad, sense. By pro^ss, we'mean all thinking 
done* to determine what the probrem is asking, what i|tformation should ^ 
be used, how a- solution might be obtained^ how a plaT)\of attack shoifld 
be implemented, and j^ow the solution might be verified. 

If a paper-and-pencil instrument could be developedjwMich would 
identify these aspects of the processes used in probLem solving,^his 
would certainly be an aid to the teacher in helping (his/her students 
improve their problen>so+ving ability. This instrument would assist 



the teacher in identi'fying processes which the student if/ould not nor- 



mally wrdte while solving the problem. Since this Instrument could be 

.administered to a large group of stujdents or to individuaf^udents,^V- J 

the teacher ^^Id be ablS^to obta.in more knowledge about the students' , 

processes by combiViing this information with hisTher observations. 

' In addition, the instrument might be used to focus the students' 

. attention on the processes used in problem solving. 

" . . ■ -# . ■ ' '"^ . >. » 

Gurova (1969) found results which indicate^! ^hat making^students 

more aWre ofxlbe reasoning processes that they 'used Improved their 

problem-solving performance. In working with f if th-^ and sixth-grade 

stutjentg, he used questions,^ apparently similar to those suggested by 



Polya, to encourage .students to analyze thelrvown thinking. He stated, 

Hone necessary factor jn the ability to solve a relatTveTy> 
complex problem requiring logical reasoning is an awareness^ 
of one's mS^tal operations in the problem^»solving process. 



Such an instruiiient might also: * • 

.» ■.. ■ . .... , .., .... '.. .. , -..../v ' ,. . ; • ... 

1. identify some approaches which students are inclined to use, 

either^efficiently or'inefficiently, and suggest approaches 

I ■ ■ • . ■ . ♦ ■' ' .■ 

: which need to be reinforced or clarified and approachfes which 

need to 'be introduced; and 

A • . / ' ^ ~" ■ ■ . 

2. y indicate growth of a 'class in certain aspects of problpm- 

K ' • \ ■■ • ' ' ■ -4 ■ ' 

■ ' solving abtlityV 



f ^ ' (5 Review of Related Literature 

To study problem- solving processes, various methods have been used 
in. an attempt to obtain observable evidence of those processes*: 
■ Patrick (1967, pp. 4-6) cited studies Which, use mechanical or manipula- 
tive d^vttes such a-s wire" puzzles, jars of watfer, and light board/, and 
t'ests which present alternatijiAes which a^low an observer to follow the 
path used to obtain a so'tution. These tasks, however, do not accurately 
represent mathematical problem sol ving since the cb(!9ces available at 
each step are more structured than ?hose avail abl^^ a mathematical 
problem-sol ving \S-ituation. , 

One instrument which has been used to assess general .problem- 
solving -ability is the Purdue Elementary ProbVem-Solvir)g' Inventory. 
(Feldhusen, Houtz, and Ringehbach, 1972). The inventory was designed 

to assess twelve skills considered: to' be .the basis Jor general problem- 

•• ■ • .? ■ ■ . ' 

solving abili.ty. These twelve skills ane: , - 

. • * , ' ^ . . ' _ ^ 

J *1^ --Sensing that a problem exists, . • ^"T"^ 

*2) defining\the problem., ^ ; ; 

3) asking question's about, t3t>e problem ^ ^ 

4) clarifying the goal in the problem situation,. \ ' ^ 

5) guessing causes, \ 
*6) noticing relevant detail^, 
*7) ttsfng familiar®bjrcts in unfamil i ail ways ^ 
*8) 'Seeing implication??^ ' - 

9) solving problems with only one solution, / * 

^ TO) "Solving problems which have'several possible solutions/ 
ll}^erify1V^ J 
; judging'n^ thef^ is sufficient information presented to 

'Solve a problem. . f 



It was^-determined that this inventory d i^ assess those skills yrtiarked 
with afl asterisk (Speedie, Hoiitz, Ringenbach, and feldhusen, W73). 
The inventory includes 47 niuiti pie-choice items in-^wl^ich situations 
are presented by means of^ models, cartoons on slides or containedNan 
the^test booklets, or Wrbal descriptions. For each situation, chi 
dren are ask6d to perform yarious tasks which correspond to the twelve 
skills mentionecl above. This inventory has been reported to measDre " 
some aspects of general problem solving and does -riot really focus on 
the measurement oK-processes that are more specific to mathematica]^ 
probler5> sol ving, ' ^ ^ ^ 

" Askl^ ^ subject to anaiyz^ his/her own thought processes used in 
solving a problem is another inethod which ha^s been used in attempting 
to determine problem-solving prpijesses, Kilpatrick (1.967, pp, 4-5) 
mentioned two ^ such app'roaches—in^ospe'ction and retrospection. Intro- 
^pection is done while the problem is being solved. The accuracy of 



this method, Wkwever, is not known and it may innibit the-^lution >^ 
process since the problem solvfer must contiijiu^Tly interrutrt himself/ 




herself to analyze his/her thinking, Retrospfectton, which asks the 

subj«!i± to recall his/her thought processes after the problem has J)een 

\solved, presents the problem of 'remembering all The ap[*i^^hes which 

were considered pr employed, . \ 

One method which does not require the subjeci to analyze h^s/her 

own thinking is to have the subject think a1ou^ as (s)he attempts to 

solve a problem. Kantowski noted (1975) some limltations^cof this method: 

. The method itself may -be unreliable sin^e an individual ''might 
remain silent during moments of deepest thought- Moreover, 
a vitalized solution CQuld be essentially different from one 
effecfted , silently- The presence of an observer cpuld^t con- - 
straints on a problem soTver in that he might not attempt solu- 
tions. which might be considered foolJsh to someone else tut 
which he^uld try if he ^ere not t?eing observed- ^. ^ 

^ • . • ■ • • ■■ r : . ' ' ' 



In spite of t-hese limitation^, it appears, t;bat'this is a viable* method* 
and it ha's ;been-- used by many in researching pnobl em §ol vlngT Many of 
those/ust/ig the^th in king-aloud method liaye" u^ed observations and audio- 
•'or,,\«deo-tap_ed: i,nterv^^ws\ fol lowed >by .^k protocol coding pro- 

c^'dM»?.e,of,'One„ fom oij^nother. This; howeyer, is tinie consuming and' • 
.''"g^ir^^tical from the standpoint of d cl-assrOom ^.teacher. 

-An ini;tial effort to develop a test which wouVa\assess one aspect 
of the mathematical ppobl em-sol ving process, has beenliiade by ,\?os,'(1976t. 
The Pr6bl em Solving Dfecision Test^focuses on ^tjiree) organizers of infor-^ 
■mation--making a. drawing, approximating artd" verifying, and cons^uciing 
• a table or chart. For -six problems, students are asked to decide which 
of these approaches could be i^sed to sol ve-tHfe problem. "^^'I* the present 
time, insufficient data i< available concerning the ef'fectivenel^ of 

I* ' " t . . . ' 

this instrument. .' • • . 

• At the present -tiie, ^istruments designed to assess -"jS^sbljem-s^olving 
pwj)cesses are efther too limited in scope or too general in Mature- to ^ 



provide Useful information concerning mathematical problem-solving. 
Other pT^edures, such as interviews, a're^ ^impractical Tor coatinual use 



'by a classroom teacher. For this reason, the Mathematical Problem 

r ■ ' ■ ' ■ ■ 

Solving Project Tjegan the, development of an instrument' which Would attempt 

,to aqcess t>ie processes used •|n mathematical problem solving and which 

would be practical for. classroom use. 

'., Development of First Instrument > \^ 

In developing the paper-and-psJcil* instrument to evaluate processes, 
an jnitial attempt was made to de)kLbp questions wh-ich. would help to >. 
Identify/progesses used by children to solve six 'problems. Before these . 
six problems we.re identified, intervviews with elementary children were " 

■ r ^ ■■ ■•- ■ ■ .... . y-" ' 



begun during the fall of 1975 to- gather information on what strate^es 
cKHdren might use while attempting to solve problems. From a fourth- 



grade class, six children were selected by the 



teacher tio provide a 



range of ability levels. After several group sessions were held to 
acquaint |5;he children with the three intervieyers and to establish an 
informal setting for the problem-Tqiying sS^ions, the interviews were 
begun on a one-to-one basis. 0|yer the course of the interv.iews, each 

child was asked -to solve twelve prpblems (see- Appendix A). The child ^ 

■. • . ■ ' ■■ • , ■ 

was asked to "think aloud" while solving the problems, and the-inter- 
viewer asked questi(^ throughout the session in an attempt to obtain 
more in^formation aboiJt any ' strategies being used and to obtain state- 
»rits which could beused^as possible responses to-questions whicH might ' 
Included In the instrument,. These interviews were ^tape recorded, 
and these recordings, along with the studenti- written work, were used- 
in diBtermlning thevsix probTems to be inclMj^ed in the instrument and; 

in develgplng the^^^^c^ ; 

• • ' li ... • ' . ■ ■ 



The ftfst instrument (see Appendix B) contained questipns corre^^; 



;j)ond1ng to the ^four. stages of the problem-sol vingf process: -comprelien- 

. . 0 ■■ * - , : ■ . 

sion, pTstnrfing, .carrying through the^plan, and Evaluation. For each 
of the six problems, the responses to the corresiponding questions were 
written primarily using two formats--one aslcing the children to answer 
yes or no to\ea(:n response aftd one- us This pro- , 

duced twelve sets of questions, two for each of the six problems. The ^ 
comprehension questions were written to determine what informtftion 
given in the problem was used in the solution and how §ome aspects 

I / : " ' - ^ . ■ ■ • - - 

of the problem were iJiter pre ted. The question concerning initial plSin- 
ning was written as a ranking question ori both forms and was identical " 



.on each siet of questions. To answer this question, the chi-ldren were 

risked' toMn^ what they did first,, second, and third after reading 

i the^prablem. To determine hoii-^he solution was obtained or sought, th'e 

next type of questions -presented several possible methods of solution. 

'■■ I ^ ■ ^ ■ . . - 

Tjhe children v^lere asked to jndica'te' which, method or methods^ were , most 

(lj|e what they^^houg^t a*bout -or what thjgy did. Lastly, the evaluation 
questions- were c(|ncernld with the checking of the solution and the de- 
gree of Confidence that t^e student had in his/her sol utionN Since all 

' V -^^ , ■ / ' ■ • ^ 

possMblg^responses could not be anticipated, mast questions included 

'S ■'■^ • ■ ■ . 

the option of writing in the correct response if it. was not included 

in the. list.' -/ , " ' 

. -.' I ' ■ -•■"^ ' " ' 

Mm-YuAtHjoutio n oj thz Inst/um^nt ■. • - . • ♦ 

The first instrument was administered-to a' class ,of 26 ohildren. 
fTnce. each child was asked to solve three problems and to apswer. three 
Bets of questions, the class was div-i(|led 'into four groups— two groups 
answermg !^es-no questions and two grqtips answering multiple-choice 
quesllonsV -as shown in Table 1 

Jble 1 



Group 


• ■ ' . .J Form 

— 1 ■ . 


Problems 




^ yes-no ' . 


AYl , AY2, AV3 


2 


^ ^ : yes-no * " 


BPi ;. BP2, BPS 


" 3 - 


* multiple choice ■ 


AWl, AWg, AW3 


4 ^' 


. mul tfple choice 

» - ■ ■ ' 


* BGl , BG2,' BG3 



' ■ After the firs"t set of problems had been randomly ^distributed, ea'ch 
child attempted to^ soVve the problem, ha nd^h is/ her work. to one of the ' 
proctorsi anci received (the corre|ponding questions." After completing 
the first , set of questions, 'the student was given the second problem, 
the second -set of questions, etc. ,'ih*a similar manner. - • 



students* responses were analyz^ed to determine whether oV not 
theii^ responses to the questions >/ere consistent vrlth ,th^ work shown 
" on the solutio*^ j|r!o and whether or^not thjeir responses were consistent' 
within - . * • .stipns,';^ T^is ^s ^ne by comp^cijjg the mfethod(s). 
indrsn^ed uj ^ ^^^^^nses^ to^'l:^ questions, with the ilfiethod(s) shown 

.on the sol Ot'ton page.- 'in^^d'ition, the- responses to items within the 

' " . ' '.-^ ■ • ■ ^ ' \ ■ , ' " 

set. of questions wh*ici> were repetitive were compared to* determine con- 

sistency. -This analysis indicated that approximsTtely 70% of the stu- 
dents answered the questions in a manner which a?pp«ared to' be consistent 
on toth counts. 

^ \An additiorml concepq was to determine what information was re- / 
vealed by the wstrument which could not be inferred from the solution 
page,*i''Some responses to comprehension questions indicated misinter- 
pretations, conditions which were ignored, arnd the use of incoV^recl in- 
formation in, the. solution of the problems. For example, one student 
gave the correct answer to the Caterpillar Problem, Problem BGl and 
BP! , by solving it incorrectly arid focusing on the wrong information. 
iShe indicated in the first comprehension question that she u^ed the - 
-^tdth of the jar, rather than its height.* By assuming that the cater- 
'pillar would climb 2 inches each day, she co^cludecTttja t it ^uld reach» 
a height of 6 Inches in 3 days. One would, not have discovered^his by 
looking only at the solution page. Responses to the compFehensioh 
questions' concerning the Candy Problem, Problem AY3 and AW3, indicated' 
that some of the conditions of this problem were ignored or misinter- 
preted. Many ^students seemed to focus on the fact that Ted received the 
^flrst and last pieces of candy and did not allow the possibility of his 
^receiving more than two, pieces. Knowing how a student had interpret?e^ 



V a problem' provided useful information in analyzing ,the]' Solution. 
■ The planning ^estion did not .^^rovide much infprfnartiog about this 
a$pect ofyth€ pr.obl em-solving process. Although m.ost oT .the responses 



seemed ccvisi stent, w^th thh Wot^t shown on the sol uttiJ^ij page and with the 
jother Pfl>-t< ofl;he;yjstr'ument, tffrv -'i-ri not provi'^le any significant tfi- 
> ' tudent^ in-i Responses to the'next o>)p<^ 

ti^, concerned with the actual sol-ution approaches, prjtf(4ded- iniorma- 
•tioh concerning apprtrarches not shown on the solution, page and explana- 
tions of calcuTati'Ons jvhich were difficult tx) interpret, t Most of those 
students who wrote only the answer on the solution page indicated that 
they had used approaches which appeared* to be consistent with their 
ansi^rs. Observations of tlje children while solving the Caterpillar 
Problem val idated the- responses in .some instances. Students who in- 
dicated that they used a ruler to solve the^problem did, in fact, use , 
a ruler. *" , 

Responses to the evaluation questions prov-ided some information 
about this phase. - However, the respjy^es cont^i/ied in the list were 
sometimes too general to give significant data. : 

Thi% analysis of the first instrumdtit indicated that students 
could^ provide some information concerning the processed used in arriving 
at a solution. However, not all' farts ^f the .instrumenf^ere considered 
to be satisfactory. FurtherV^rk on the instrument was guided by the 
■followl^ng conclusions:" 

N - The comprehension questions .were. appropriate and provided 
necessary Information in- analyzing the solution. 

- The^^uestioos regar'ding sblution approaches were informative, . 
yet they,dijdjj£>e^low students to indicate all methods*used. 

^~ Tte .evaj/dation questions were too general and somewhat inappro- . 



^ ■ .... .. ' . , J 4 



Y 



^ ^ ' ) Development of Secppd Instrgment 

-^^ , , . ; y \ ^ \ 

\ ^ In an atteihfJt to determine if more> information' regarding processes 
douTd>||^e obtained through^^he use of, a paper-and-pencil instrument, a^ 
revision of the questions (Joncprning brie-of. the six problems, the Cater- 



pillar Problem,* was begurt. This ptfablem was Wiected because ofyi^'be 
^variety thods tha(€ ctiijdren had n<pd in atttempting to so^Ve it 
" and awareness of those methods that chi idren exhibited. 

' - . ,. ■ — ■ ■ - ■ . • . , 

In the firstr tria|j'^ some /ambiguity was found in the problem state- 
ment. In the second versioA (see Appendix C), *the jar has a lid. This 
made it physically impossible for the caterpillar to climb beyond 8 
^inc)Ti|§^ Also the width .of the jar was changed from 6 inches to 7 inches 
^ It would then be more. apparent if the student focused on t^e width since 
7 inches would not appear- at any stage of ' a. correct solution approach. ' 
Seconrf tyutmmzyit 

— : ^ : ^ . ■ 

. •The format of questions on the -second instrument (see Appendix C) 
' was, modified multiples-choice, where children were asked to check one 
''•or more of the possible resporises. Again, tThe option was provided in 
^ some of the questions to write in responses other than thqse listed.^ 

Although this instrument contained questions corresponding to each 
of the four stages -of the problem-solving process as did the first 
insjtrumerrlfi the K)rganizatioh was somewhat different following the first | 
page. ^The first questions concerning comprehension were left essential: 



unchanged. The only differences were^those necessitated by the change^ 
in ^he problem statement. . > ^ * ^ 

The next tvp pages pbntailied questions regarding the approaches 
or strategies that were used l)y the Qhilbren. The first of these two 
pages displayed pictures and calculations representing four different 



;vapproaCj^es -which children have used in attempting to solve the problem. , 
Thfe children were ajked to, select' from the 1%ur approaches or to des- 
cnbe a fifth to indicate what they Vad.tione' while attempttn^g^o solve 
'tha probl«n--what- they had thought about, what they had dane'Virst, and 
yhat t1i^ had done to get an answer\^he first^ instrument^ had merely'j^ 
asked'^th^ children to indicate how tKe[y had attempted to S(?lve 1;he 
probl-emy It wd« hopejl that this refinetfient would. allx)w the children ^ ; 
to- identify ntethods they had con^ered or employed during various stages 

' of the sol^jtioiVprocessv/' ^ ■ * , 

On the nex* page was a list of responses which indicated various . 
strategies."' Same statements reiteraled. methods' shown on the previous^ 
page and served, as a reliability ch6ck; others were ,i hql uded to bbtafn 
n»re information- about mental processes.* « 

The fourth^-page was, concerned^with initial planning. The children ' ^ 
were asked to. mark the statement (s) which described' what they did as 
soon as they had rea'd the problem. It was intended that their choices 
would indicate whether the student attempted -a plan of atfack .ininediately 

' aftej^readlng the problem'^or /equired a certain aw)unt of "messing around" 

time. The first instrument. ha(k contained some of these statements and- - - 

others undefthe.pl anning question. These statements were divided and 

expanded in the second instrument," Those responses which r^j^resented 

strategies- which might have been used during later stages of tfie solui- ^ 

tion process were "included on the previous page. 

K. . , ' ' '' ^ 

On the next page were ,quesJ:1ons to determine not only' if the stu-' 

dent attempted to solve the probligv using one or more than one method, 

but also why one metljod was decidedi-to be sufficient or insufficient. 

These -questions were added to^the second instrument to determine further 

the children's awarene^ of .the various methods that thej(^ad employed 



The final. set ofvquestions asked the cfiil-^ren to. indkate whether 
or riot they 'had checked their answers and how the ^Jiecl^ing was^ne or 
why th6y decided not to check. Thiis differed from the f '^ » '».-iMrMrTion» 
ri^ gijJy askpd '^^the student had checked the solution. ^ . 

y-T — ^ — ^ ^ . ' ' 

V This second instrument was administered by^^ive interviewers to 



32 fourth-, fifth-, and sixfh-grade childre^ in the Oakland County school 
^district i^r\ Michigan/ HTtfese cf^Hdren were selected from experimental 
classes, classes which used the MPSP materials, and frqm contrpT^assesS 
Children were also selected from, high-, average-, and low-ability groups' 
determined by combined pretest scores on the Stanford Achievement Test 
and a problem-solving scale, X023, used in the National Longitudinal 
Study of Mathematical Abilities. After attempting to splve two other 
problems, the children were asked to solve the Caterpillar Problem while 
doing their thinking aloud* Aft^r finishing their work on the problem, 
the children completed the set of questions concerning the Caterpillar, 
Problem. These interviews were tape recorded and these r^ordings were 
used/in the analysis of the results* 

An analysis of the students' ^responses on the second instrument 
was performed in the same manneir as the analysis of the first ins*tru- 
ment* In addition, the tape recordings were used to validate the con- 
sistency of the responses to the questions wi^jj the solution page and 
the consistency of the responses within the instrument** This analysis 
Yndicated that^pproximately 45% of- the students responded in a manner 
Which was considered consistent in both respects, approximately 30% 



r reippnded in a manner which sometimes appeared to be consistent^, and 
approximately 25% responded in a manner which appeared, to be almost 
.V. totally inconsistent ojr, arbitrary. An analysis of these three groim- 
of students will be presented to 111u«:t-''',e ' m^u nd weak- 
of the jocund instrument. ' 
; ■ The. responses which appfeaml ,to be consistent either verified whai 
. one mighf.have inferred from the solution page or provided additional^ 
information which further explained the work shown o«r\he' solution page. 
One student made a drawing which resembled a ruler on the solution. She 
indicated on the instrument that this had been used at^ first, but she 
reached the solution by\dding and subtracting mentally. The recording 
of the interview verified that the solution page showed only the pic- 
ture which helped her to understand the problem and that the instrument 
accurately indicated the processes used to reach the solutiorv. 

Another example is shown by the first set of responses in A^ppendix 
D. Susan's solution page shows some calculations/which alone do no.t. 
indicate a clear solution method. ^h>vclarified this somewhat on page 2 
of the instrument by marking thV two methods that were similar to what 
she Ha.d done and by writing a more organized set of calculations. She 
indicated on page 5 that she constffered these methods as ^ne approach, 
rather than 5;eparate approaches. tK^ recording of the interview showed 
that this was, the case (see Appendix*D); 

Jhe instrument does not give>p^|^lete an explatiaJtion of Susan's 
thinktng as does the interview, but ^t^does give a more complete picture 
than Is shown on the solution, page".- 'f^ ' .- . 

Other stj^dents, whil^ providing some information about their method 
of solutfon, were not as consiAent as those mentioned above. The second 
set of responses found in Appenclix-.D provides. an example. Eddie marked 



*on page 2 that he fir-st thought of a rule^ hn: Miitr i ^et 
this answpr. He al^ "n ' • f . r.- . u ni'n;>< days 

and cnecKed to see if the number was cori^ipt/ The recording of the 
interview verified this (see Appendix D), ' . • 

His resporrses on page 5, however, were not consistent. 'He marked 
that the first tjme, he tried to solve the prSblem, he knew he wa^ right, 
•Yet he also mai^ked that he tried a)seco^^"^y because' he ^asn't^re 
* that the first way was right, 

When students gave both consistent >B4^nconsi stent responses, most 
of the Incortsistencies' were found on pages 5 and 6, These conflicting 
responses regarding confidence in and checking of the answer may, have 

/ . 

resulted, J n part, from' misinterpretations of the questions, 

O^her students gave responses to the questions which were almost 
totally inconsistent and provided little or no information concerning , 
their method of solution. An example of this is pro-vided .fey the third 
example in Appendix D. .Trudy ifidicated on page 3 that she mjid^ a drawing, 
*yet no drawinig was shown on the solution page. On pages 5 and 6, she 
indicated that she checked her answer and saw that the first way was 
wrong, and that she did not check her answer. The recording of the 
interview seemed^ equally confusing (see Appendix D), 

Other students responding in this manner gaVe conflicting responses 

■ - (J 

'to the comprehension and approach questions. Some indicated that they 
did jfet^iise information which was Used on the solution page, Otheirs 
marked that they used almost every possible approach while solving the . 
problem. These student^ended to mark many more response? than those 
Students who appeared to be consistent, * 



n In tnii, ;a .e, the instrument joes not give an accurate pictur^'^of 
'.the solution .process. However, one (kies^not require the interview to 
discover this. The. interviews 'did v^ifify that those papers which did 
irgt provide useful information concerning problem-so^lving prbcesses 'tould 
be determined 'by the inconsistent responses found in the instrument. <^ 
The analysis of student responses and the interviews allow several^ 

conclusions to be «lrawn: . . 

. ^"^'^ ■ ♦ • . ' * : 

s- Th'e instrument can pro\Ade information which- explains uhcleajy < 
^ procedures found. on the solution page and additional informa- 
tion concerning approaches which were not written on the solu- • 
tion p*gg. \ . 

.. ■ ' ■ ■ ■, ^' ■■ / . ■■ : . . ■ 

- The instrument provides sufficient means of dete^ining the 
reliability of the responses. Erroneous or arbitrary responses 
can be discovered byi'examining those, responses to items which 
should be consistent. * 

- Responses to questions contained on the first four pages of 
the instrument indicate tfjat these questions are correctly 
interpreted by most students and that they a^e appropriate 
with respect to the comprehension, planning, and solution , 
stages of the problem-solving pr^^ss. 

- The misinterpretation? of the q^Jestions on the final. two pages 
of the instrument indicate that these questions netd to be re- 
worded or presented-^using a different format. \ 

^ . • Suggestions for furt1:ier Inyestiqatioli ' 

Although there are some weaknesses in the->*weP instri/ments whicli 
have been described^ it appears that they'do provide Ijseful information 
concerning the •processes used by children to solve mathematical -prob- 
lems. However, further investigation and refinement is' suggested by 
the results of the second adminjjStr^ation. . ^ 

First, additional questions might be included.- General impressions 
concerning the various approaches which children used were obtained;^ 
however, more specific questions are nedes^ry to identify the approaches 
which are considered and- the reasons for accepting o> rejecting, * par-v- 
ticular approach. In addition,, questions concerning preferences of 



approaches or strategics might be included. If responses to these 
questions are consistent over a periofi of time, these may provide in- 
sight into a student 'sfproblem-solving style.^ ^ ^ 

Secondly, the organization of the iiTstrument requires seme altera- 
t1on. In the first instrument ,(^'he questions proceeded in a /rather ,1. 
systematic manner through the vaji^ious ^|tages of the pro W em- solving/^ 
process. The questions w«fre ordep^ yi much the same way as a stutlent 
might have orderell his/her owrf thinking.'* In the secoarLinstrumeht , it 

-^appears; 'that the student was asked to, consider the so^trE>^ pn^cess in 
a nonchronologicat order, . The^ student *i/as asked fo indicate what in- [ 
formation was us6d, what was thought about first, what was done to get 
an answer^ what was done immediately after reading the problem, etc, 
This may have been confusing to some students, ^ ^ 

If this instrument can be refined and developed fo\?-j^ther problems, 
it should be a valuable tool for the classroom teacher. Since many 
of the items could be identical for all -problems , it could be easily 
adapt^ to^ny problems which the teacher might wish to use. It. appears 
that it has ^ea^potential as a diagriWtic instrument, to assist in 
determining which ty^e^ of problems pose difficulties for 3 student 
and what aspects of a particular problem prevent the student from solving 
it. The refinement of this instrument should be a useful aid to teachers 

dn assisting chtjclren to become better problem solvers, • 

. ■ . ^ ■ .. ■'. .) ■ 
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